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Objectives  of  the  Annual  Assessment  of  Major 
Weapon  System  Programs 


•  Provide  a  cost/schedule  snapshot  of  DOD’s  2008  portfolio  of 

major  weapon  system  programs  and  a  comparison  to  portfolios  at 
two  other  points  in  time  -  1  year  ago  and  5  years  ago 

•  Provide  observations  about  the  portfolio’s  balance,  performance 
of  newer  programs,  and  ability  to  deliver  to  the  warfighter  on  time 

•  Analyze  outcomes  and  knowledge  attained  at  key  junctures  in 

the  acquisition  process  for  a  subset  of  the  47  programs  primarily 
still  in  development 

•  Gather  data  on  other  factors  that  might  impact  program  stability 
and  outcomes  such  as:  cost  estimating,  requirement  setting, 
software  management,  and  program  office  staffing 

•  Provide  an  update  on  DOD  acquisition  policy  changes 
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Snapshot:  Committed  and  Planned  Spending  on 
Current  Portfolio  of  96  Programs 


Billions  of  FY  2009  dollars 
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Snapshot:  Overall  Cost  and  Schedule  Growth  for  the 
Current  Portfolio  of  96  Programs 


Performance  of  DOD’s  Major  Defense  Acquisition  Program  Portfolio 


Portfolio  status 

Fiscal  year  2003 

Fiscal  year  2007 

Fiscal  year  2008 

Number  of  programs 

77 

95 

96 

Total  planned  commitments 

$1.2  trillion 

$1.6  trillion 

$1.6  trillion 

Commitments  outstanding 

$724  billion 

$875  billion 

$786  billion 

Change  to  total  RDT&E  costs  from  first  estimate 

37  percent 

40  percent 

42  percent 

Change  in  total  acquisition  cost  from  first  estimate 

19  percent 

26  percent 

25  percent 

Estimated  total  acquisition  cost  growth 

$183  billion 

$301  billion 

$296  billion 

Share  of  programs  with  25  percent  or  more  increase 
in  program  acquisition  unit  cost 

41  percent 

44  percent 

42  percent 

Average  delay  in  delivering  initial  capabilities 

18  months 

21  months 

22  months 

£oyraarttpAQ\analysis  of  DOD  data. 
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Observation:  Top  10  Programs  Continue  to  Strain 
DOD’s  Buying  Power  Elsewhere 


•  10  of  the  department’s  largest  programs,  commanding 
about  50%  of  the  acquisition  dollars  in  the  portfolio, 
have  experienced  significant  cost  growth  and  quantity 
reductions: 


•  Development  costs  have  grown  by  32% 

•  Total  program  costs  have  grown  by  12% 

•  Overall  quantities  have  been  reduced  by  32% 

•  7  have  acquisition  unit  costs  of  greater  than  40% 
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Observation:  Top  10  Programs  Continue  to  Strain 
DOD’s  Buying  Power  Elsewhere 


Total  cost 

(fiscal  year  2009  dollars 
in  millions) 

Total  quantity 

Acquisition 
unit  cost 

Program 

First  full 
estimate 

Current 

estimate 

First  full 
estimate 

Current 

estimate 

Percentage 

change 

Joint  Strike  Fighter 

206,410 

244,772 

2,866 

2,456 

38 

Future  Combat  System 

89,776 

129,731 

15 

15 

45 

Virginia  Class  Submarine 

58,378 

81,556 

30 

30 

40 

F-22A  Raptor 

88,134 

73,723 

648 

184 

195 

C-17  Globemaster  III 

51,733 

73,571 

210 

190 

57 

V-22 

38,726 

55,544 

913 

458 

186 

F/A-18E/F  Super  Hornet 

78,925 

51,787 

1,000 

493 

33 

Trident  II  Missile 

49,939 

49,614 

845 

561 

50 

CVN  21  Nuclear  Aircraft  Class  Carrier 

34,360 

29,914 

3 

3 

-13 

P-8A  Poseidon  (MMA) 

29,974 

29,622 

115 

113 

1 
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Observation:  Promised  Capabilities  Continue  to  Be 
Delivered  Later  Than  Planned 


Schedule  Delays  for  DOD’s  2008  Program  Portfolio 


Programs  planning  to  achieve  IOC 
on  time  (or  less  than  1  month  late) 
(20  programs) 


Programs  planning  to  achieve  IOC 
between  1  to  12  months  late 
(17  programs) 


Programs  planning  to  achieve  IOC 
between  1 3  to  24  months  late 
(13  programs) 


Programs  planning  to  achieve  IOC 
between  25  to  48  months  late 
(12  programs) 

Programs  planning  to  achieve  IOC 
more  than  48  months  late 
(10  programs) 
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Observation:  New  Programs  Are  Performing  Better 
Than  Older  Programs  at  This  Time 


Changes  in  Program  Cost  and  Schedule  by  Age  of  Program 
Fiscal  Year  2008  Portfolio 


Change  in  total 

Change  in  total 

Age  of  Program 

RDT&E  costs 
from  first 
estimate 

acquisition  cost 
from  first 
estimate 

Average 
change  in 
quantities 

Average 
number  of 
months  late 

Number  of 
programs 

15  or  more  years  since 
development  start 

47  percent 

19  percent 

-39  percent 

37  months 

10 

10  to  14  years  since 
development  start 

73  percent 

53  percent 

52  percent 

26  months 

17 

5  to  9  years  since 
development  start 

37  percent 

31  percent 

9  percent 

22  months 

25 

Less  than  5  years 
since  development  start 

12  percent 

1 1  percent 

1  percent 

5  months 

28 

fioyraarttpAQ\analysis  of  DOD  data. 
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Caveat:  Historically,  Largest  Percentage  of  RDT&E 
Cost  Growth  Occurs  After  CDR 


Percentage  of  RDT&E  cost  increase  over  development  estimate 


19.7% 


r  8.5% 


Note:  Does  not  add  due  to  rounding. 
Source:  GAO-06-391 
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Knowledge  Analysis:  Newer  Programs  Are  Starting 
with  Higher  Technology  Readiness  Levels 


•  Since  2003,  there  has  been  a 
significant  increase  in  the 
percentage  of  critical 
technologies  at  least  nearing 
maturity  (demonstrated  in  a 
relevant  environment)  prior  to 
development  start. 


•  In  the  last  3  years,  all  5 
programs  entering  system 
development  had  their  critical 
technologies  demonstrated  in 
at  least  a  relevant  environment, 
in  accordance  with  the  DOD 
and  statutory  criteria. 


Percent 


(10)  (12) 

Year  program  entered  system  development 


(5) 


J  Mature 

J  Nearing  maturity 


§5*lr4^2©€©ana|ysis  dod  data. 


Note:  Number  of  programs  and  technologies  in  parenthese|qQ 
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Knowledge  Analysis:  Programs  Report  More  Design 
Drawings  Complete  at  CDR 


•  Since  2003,  the  average 

’  ^  Percent 

percentage  of  design  drawings  ™ 

releasable  for  programs  at  the 
critical  design  has  steadily  80 

increased. 


60 

•  However,  designs,  on  average, 

are  still  far  from  stable  and  « 

concurrent  technology 
development  increases  risk  of  20 

subsequent  design  changes 
and  rework. 

0  2003  2004  2006, 2007, 

or  prior  or  2005  or  2008 


^^^Stable  design 

64.85% 

(16) 

57.93% 

(8) 

39.19% 

(4) 

Year  of  critical  design  review 


§7W2©f9?)analysis  data. 


Note:  Number  of  programs  in  parentheses. 
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Knowledge  Analysis:  Programs  Conducting  Early 
Systems  Engineering  Have  Better  Outcomes  to  Date 


Early  systems  engineering, 
ideally  before  a  program  enters 
development,  is  critical  to 
ensuring  that  requirements  can 
be  met  with  available 
resources. 

Programs  that  conducted  key 
systems  engineering  events 
prior  to  development  start  have 
experienced  lower  cost  growth 
on  average  and  often  have 
shorter  delays  in  achieving 
initial  operational  capability. 


Average  RDT&E  Cost  Growth  by  Timing  of  Key 
Systems  Engineering  Reviews 

Percent 


SRR  SFR  PDR 


Programs  that  held  the  review  before  development  start 
Programs  that  held  the  review  after  development  start 
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Data  on  Other  Factors:  Requirements,  Software  Mgmt, 
Cost  Estimating,  Staffing 


•  Programs  that  changed  key  system  requirements  after 
starting  development  had  cost  increases  3x  greater  than 
others  and  schedule  delays  2x  greater 

•  Programs  with  software  growth  of  25%  or  more  have 
experienced  much  more  cost  and  schedule  growth  then 
those  with  lower  levels 

•  Most  programs  initial  cost  baselines  are  based  on  the 
program’s  or  service’s  own  estimate  and  assumptions  rather 
than  on  an  independent  estimate  done  by,  for  example,  the 
CAIG 

•  Most  programs  are  not  now  able  to  fill  all  government 
positions  (PM,  SE,  K,  Admin)  they  have  been  authorized 
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Areas  of  Relative  Agreement  Between  the  DOD  and 
GAO  Concerning  Problem  Sources 


•  Acquisition  problems  have  their  roots  in  the 
requirements  and  funding  processes 

•  Programs  are  initiated  with  poor  foundations  and 
inadequate  knowledge  for  developing  realistic  cost 
estimates 

•  Programs  move  forward  with  artificially  low  cost 
estimates,  optimistic  schedules  and  assumptions, 
immature  technologies  and  designs,  and  fluid 
requirements 

•  Imbalance  between  needs  and  the  resources  available  to 
meet  them  contributes  to  budget  and  program  instability 

•  Changing  or  excessive  requirements  cause  cost  growth 
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Recent  DOD  Acquisition  Policy  Changes  Have  the 
Potential  to  Significantly  Improve  Outcomes 


More  discipline  in  the  early  phases  and  throughout  process  will  put  more 
knowledge  (less  risk)  upfront  in  the  process 

•  Materiel  Development  Decision  required  for  all  programs  to  level  requirements  and 
resources  before  starting  the  program 

•  Configuration  Steering  Boards  established  to  control  requirements  creep 

•  Preference  for  incremental  development,  with  baselines  for  each  increment 

•  Preference  for  holding  PDR  before  system  development  start 

•  Competitive  prototyping  required  prior  to  Milestone  B  as  part  of  technology 
development  phase 

•  Growth  of  Capability  Portfolio  Managers  to  elevate  programs  from  stovepipes 
(reduce  gaps/redundant  programs) w 

Areas  of  continuing  concern 

•  Still  does  not  require  “time  certain”  development 

•  Still  allows  for  concurrent  technology  and  product  development  and  production 

•  Post-CDR  assessment  still  not  a  milestone  decision 

•  Controls  not  in  place  to  ensure  accountability  and  adherence  to  intent  of  new  policy 
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•  Acquisition  policy  initiatives  are  aimed  at  the  proper  target  -- 
the  front-end  of  the  process  and  the  systems  engineering  that 
is  required  there 

•  There  is  growing  consensus  that  the  root  causes  of  poor 
outcomes  lie  in  misunderstood  requirements,  unreliable 
estimates,  and  unmanageable  development  times 

•  The  Congress’  reform  legislation  also  targets  the  appropriate 
troublemakers  -  lack  of  systems  engineering  expertise,  lack 
of  accountability  and  independence  among  key  players 

•  What’s  doable  on  paper  is  not  always  doable  in  reality  -  the 
people  involved  in  this  enterprise,  all  of  them,  must  be  willing 
to  change  the  way  we  develop  and  deliver  weapon  systems 
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Observation:  Top  10  Programs  Continue  to  Strain 
DOD’s  Buying  Power  Elsewhere 


RDT&E  and  Procurement  Funding  2009-2013 

(FY09  Dollars) 


B?lr4?2©t^ana|ysis  dod  data. 
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Knowledge-Based  Acquisition  Model  Focuses 
on  Retiring  Risks  by  Key  Decision  Points 


Material  Development 
Decision 


Knowledge 

Based 

Model 


5  to  6  years  or  less 


Development  Start 
B 


Production  Start 


Technology 

development 

Product  development 

Integration  Demonstration 

Production 

Of  o 

PDR  |  CDR 

i 

Knowledge  Point  1 
Technologies,  time,  funding  and 
other  resources  match  customer  needs. 


Knowledge  Point  2 
Design  is  stable  and  performs 
as  expected. 


Knowledge  Point  3 
Production  meets  cost,  schedule, 
and  guality  targets. 


Decision  to  invest  in  product  development. 


Decision  to  start  building  and  testing  Decision  to  produce  first  units  for 

production  representative  prototypes.  customer. 


•  Model  provides  framework  for  incremental,  time  certain  (development  constrained  to  5  to  6  years 
or  less),  and  knowledge-based  approach  to  weapon  system  acquisitions. 

•  Success  requires  structured,  disciplined  application  and  adherence  to  model. 

•  Knowledge  points  align  with  key  investment  inflection  points. 

•  Controls  are  in  place  for  decisions  makers  to  measure  progress  against  specific  criteria  and 
ensure  managers  capture  key  knowledge  before  moving  to  next  phase. 
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Other  Factors:  Programs  that  Changed  Key 
Requirements  Experienced  Added  Instability 


•  For  programs  that  had  at  least  one  requirements  change,  the 
average  RDT&E  cost  was  more  than  three  times  higher  and  the 
average  schedule  delay  was  twice  as  long  as  programs  without 
these  changes. 


Percentage  ot  change  in  RDT&E  cost 


requirements  requirements 

changes  changes 


Months 


requirements  requirements 

changes  changes 


fioyraarttpAQ\analysis  of  DOD  data. 


20 


Accountability  *  Integrity  *  Reliability 


Big  Picture:  DOD  Investment  Remains 
High,  Most  Likely  Unsustainable 


Fiscal  year  2009 
dollars  in  billions 


Research,  Development,  Test  and  Evaluation  and  Procurement  Funding 


fioyraarttpAQjanalysis  of  National  Defense  Budget  Estimates  for  the  Fiscal  Year  2009  Budget. 
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